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HYPERBOLIC SPACE 



SUSANNA DANN 

Abstract. The Busemann-Petty problem asks whether origin-symmetric 
Q-i convex bodies in R" with smaUer central hyperplane sections necessarily 

have smaller volume. The answer is affirmative if n < 4 and negative if 
n > 5. We study this problem in the complex hyperbolic n-space HJi and 
prove that the answer is affirmative for n < 2 and negative for n > 3. 
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Introduction 

The Busemann-Petty problem asks the following question. Given two origin 
symmetric convex bodies K and L in M" such that 

Vo\n-i{K nH)< Vol„_i(L n H) 

for every hyperplane H in M" containing the origin, does it follow that 

Vol„(K) < Vol„(L)? 

The answer is affirmative if n < 4 and negative if n > 5. The problem, posed 
in 1956 in [7], was solved in the late 90's as a result of a sequence of papers 
[27, 3, 14, 5, 28, 29, 9, 10, 34, 16, 17, 35, 12], see [22], p. 3-5, for the history 
of the solution. 

Since then the Busemann-Petty problem was studied on other spaces as were 
its numerous generalizations. V. Yaskin studied the Busemann-Petty problem 
in real hyperbolic and spherical spaces, [33]. He showed that for the spherical 
space the answer is the same as for M", but not so for the real hyperbolic 
space, where the answer is affirmative for n < 2 and negative for tt, > 3. A. 
Koldobsky, H. Konig and M. Zymonopoulou demonstrated in [24] that the 
answer to the complex version of the Busemann-Petty problem is affirmative 
for the complex dimension n < 3 and negative for n > 4. Other results on the 
complex Busemann-Petty problem include [37, 38, 30, 23]. 
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In this article we consider the Busemann-Petty problem in the complex 
hyperbolic n-space. For ^ G C" with |^| = 1, denote by 

n 
k=l 

the complex hyperplane through the origin perpendicular to ^. We identify C" 
with R^" via the mapping 

(6l + ^62, ...,^nl+ i^n2) H- (^11, ^12, . . . , inl, Cn2) ■ (1) 

Under this mapping the hyperplane turns into a {2n — 2)-dimensional sub- 
space of ]R^" orthogonal to the vectors 

C — (Cii) Ci2, ■ ■ ■ , Cm, Cn2) and ^± — {—^12, Cll, ■ ■ ■ , — Cn2, Cnl) ■ 
A convex body K in R^" is called Re -invariant, if for every 9 e [0,27r] and 
every ^ = (61, 62, ■ ■ ■ , U, U) e 

M\\k = 1 1 -Re (61, 62), ■ ■ ■ , Rd{Cnl,Cn2)\\K , 

where Re stands for the counterclockwise rotation by an angle 9 around the 
origin in M^. 

Recall that origin symmetric convex bodies in C" are unit balls of norms on 
C" and therefore, under the mapping (1), they are i?^i-invariant convex bodies 
in R^'*. We will work with the ball model of the and consequently we will 
only consider bodies contained in the unit ball. We denote the volume element 
on by dfin and the volume of a body K in M^" with respect to this volume 
element by HVol2n(K) to distinguish from the Euclidean volume of K. 

Now the Busemann-Petty problem in can be posed as follows. Given 
two i?6)-invariant convex bodies K and L in R^" contained in the unit ball such 
that 

HVol2„-2(^ n H^) < HVol2„_2(i^ n H^) 
for C an element of the unit sphere S^"~^ of M^", does it follow that 

HVol2n(i^) < HVol2n(i:)? 

Analytic solutions of the Busemann-Pctty problem in different settings are 
based on establishing a connection between a certain distribution and the 
problem. E. Lutwak introduced a class of intersection bodies in [28] and es- 
tablished a connection between this class and the Busemann-Petty problem 
on M". Recall that an origin symmetric star body K in is an intersec- 
tion body if and only if || ■ is a positive definite distribution on W\ see 
Theorem 4.1 in [22]. Later, A. Zvavitch solved the Busemann-Petty problem 
on M.^ for arbitrary measures, [36]. He hnked the problem to the distribution 

where /„, a locally integrable function on R'*, is the density 

function for a measure on R" and fn-i, a function on R" integrable on 
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central hyperplanes, is the density function for a measure /Xn-i on M"~^. In 
[33] V. Yaskin established a connection between the Busemann-Petty prob- 
lem in hyperbolic and spherical spaces and the distributions j^_|_(|],j||^|-i-)2 ^ ^ 
special case of Zvavitch's theorem. Recently, A. Koldobsky at al. found a con- 
nection between the 2-intersection bodies and the Busemann-Petty problem 
on C", [24]. The classes of /^-intersection bodies were introduced in [18, 19]. 
Recall that an origin symmetric star body K in M" is a /c-intersection body, 
< k < n, a and only if || ■ ||^'^ is a positive definite distribution on M", see 
[19]. This is also the case for the Busemann-Petty problem on W^, namely 
we prove in Theorem 3 that the answer to the problem is affirmative if and 
only if for every i^^-invariant convex body in M^" contained in the unit ball 

||j.||-2 

the distribution — " "k jg positive definite. Then we prove that the latter 

is true for n = 2 and false for n > 3, providing a solution to the problem. In 
our proof we use methods from [33], [24] as well as recently obtained results 
for complex star bodies from [25] . 

A few generalizations of the complex Busemann-Petty problem have been 
considered so far, sec [37, 38]. For another result in the complex hyperbolic 
space related to convex geometry see [2]. Some other extensions of results 
from convex geometry to non- Euclidean settings include [8, 11, 4, 1]. For other 
generahzations of the Busemann-Petty problem see [6, 18, 19, 20, 21, 36, 26]. 



1. Preliminaries 

1.1. Complex Hyperbolic Space. The material of sections 1.1.1 and 1.1.2 

is taken from the book by Goldman [15]. We refer the interested reader to this 
book for more information. 



1.1.1. The Ball Model. Let F be a complex vector space. The projective space 
associated to V is the space F{V) of all lines in V, i.e. one dimensional complex 
linear subspaces through the origin. 

Let C"'^ be the (n-l-l)-dimensional complex vector space consisting of (n+1)- 
tuples 

" Z' 

Zn+l 

and equipped with the indefinite ^ Hermitian form 



{Z, W):={Z',W')-Z^+^Wn+i 

= Z^Wi + • • • + ZnWn - Zn+lWn+1 , 



neither positive- nor negative-semidefinite 
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where Z' is a vector in C" and G C. Consider the subset of negative 

vectors ofC^'^ 

N :^{Z e C"'^ : {Z,Z) < 0}. 
The complex hyperbolic n-space is defined to be P(iV), i.e. the subset of 
P(C"'^) consisting of negative hues in C""'^. We identify with the open unit 
baU 

B'* := e C" : {z, z) < 1} 
as foUows. Define a mapping A by 

A : C" — > P(C"'^) 



z' 



z' 



Since for negative vectors of C"'^ the {n + l)-coordinate is necessarily different 
from zero, C A(C"). The mapping A identifies B" with and 9B" = 

g2n-l ^ £n ^i^j^ ^jjn^ 

Theorem 1. ([15], Theorem 3.1.10) Let F C P(C"'^) he a complex m-di- 
mensional projective suhspace which intersects Hg. Then F n H^^ is a totally 
geodesic holomorphic submanifold biholomorphically^ isometric to HJ?. 

The intersection of with a complex hyperplane is a totally geodesic holomor- 
phic complex hypersurface, called a complex hyperplane in Hg. Its boundary 
is a smoothly embedded {2n — 3)-sphere in dM.^. 

1.1.2. The Bergman Metric and the Volume Element. We normalize the Berg- 
man metric, a Hermitian metric on Hg, to have constant holomorphic sec- 
tional curvature —1. It can be described as follows. Let x,y he a pair of 
distinct points in B" and let 'fe^ denote the unique complex line they span. 
The Bergman metric restricts on 'fe^ fl B" to the Poincare metric of constant 
curvature —1 given by: 

AR'^dzdz 
(i?2 _ ^2)2 ' 

where it! is the radius of the disc 'fe^ fl B" and r — r{z) is the distance to the 
center of the disc 'fe^ fl B". As 'fe^ is totally geodesic, the distance between 
X and y in HJ^ equals the distance between x and y in fl B" with respect 
to the above Poincare metric. Moreover, the geodesic from x to y in is 
the Poincare geodesic in n B" joining x and y. The Poincare geodesies are 
circular arcs orthogonal to the boundary and straight lines through the center. 
The volume element on is 

d^. = 8'^.^""^^^'^ 



biholomorphic mapping = conformal mapping 
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where da is the vohime element on the unit sphere S^""^. 
1.2. Convex Geometry. 

1.2.1. Basic Definitions. The main tool used in this paper is the Fourier trans- 
form of distributions, see [13] as the classical reference for this topic. As usual, 
denote by the Schwartz space of rapidly decreasing infinitely differen- 
tiable functions on R", also referred to as test functions, and by »S'(R") the 
space of distributions on R", the continuous dual of 5(R"'). The Fourier trans- 
form / of a distribution / is defined by — (/? 'i^') every test function 
(fi. A distribution / on M" is even homogeneous of degree p e M, if 

(/(x),^(^)) = |ar^ 

for every test function (p and every a e M, a 7^ 0. The Fourier transform of an 
even homogeneous distribution of degree p is an even homogeneous distribution 
of degree —n — p. We call a distribution / positive definite, if for every test 
function ip 

{f{x),ip*p{-x)) > 0. 

This is equivalent to / being a positive distribution, i.e. > for every 

non- negative test function ip. 

A compact set K in is called a star body if every line through the origin 
crosses the boundary in exactly two points different from the origin, and its 
Minkowski functional is defined by 

\\x\\k '■— min{a > : x & aK} . 

The boundary of K is continuous if || • \\x is a continuous function on M". If 
in addition K is origin symmetric and convex, then the Minkowski functional 
is a norm on W. A star body K is said to be k-smooth, A; e N U {0}, if the 
restriction of || ■ \\k to the unit sphere S"~-^ belongs to the class C'^(S""^) of 
k times continuously differentiable functions on the unit sphere. If || • H^- e 
C'^(S"~^) for any k E N, then a star body K is said to be infinitely smooth. 
For X E S^^^, the radial function of K, Pk{x) = is the Euclidean 

distance from the origin to the boundary of K in the direction x. For all 
bodies considered in the sequel, the origin is an interior point. 

1.2.2. Approximation Results. One can approximate any convex body K in 
in the radial metric 

p{K,L):= max Ipxix) - pl{x)\ 

by a sequence of infinitely smooth convex bodies with the same symmetries 
as K, see Theorem 3.3.1 in [31]. In particular, any i?0-invariant convex body 
in R^" can be approximated by infinitely smooth it^^i-invariant convex bodies. 
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Any /c-smooth star body K can be approximated by a sequence of infinitely 
smooth star bodies K^n so that the radial functions converge to pk in the 
metric of the space C'^(S"~^), see [22], p. 27, preserving the symmetries of K 
as well. 

1.2.3. Fourier Approach to Sections. It was shown in [22], Lemma 3.16, that 
for an infinitely smooth origin symmetric star body K in R"^ and < p < n, 
the Fourier transform of the distribution 11 X 1 1 IS clll infinitclv smooth function 
on M"\{0}, homogeneous of degree —n+p. We shall use the following analogue 
of the Parseval's formula: 

Lemma 1. ([22], Lemma 3.22) Let K and L he infinitely smooth origin sym- 
metric star bodies in R", and let < p < n. Then 

[ (II • ll/)"(^)(ll • IIZ"^")"(^)^^ = (27r)" / ll^ll/ll^llZ'^^'c?^- 

We will also use the following version of the above lemma. 

Corollary 1. ([22], Corollary 3.23) LetkeN with < k < n. Let f and g be 
two even functions on R", homogeneous of degree —k and —n + k, respectively. 

Suppose that f represents a positive definite distribution on R" and let fio be 
the finite Borel measure on S**"^ that corresponds to f by Corollary 2.26 in 
[22]. Then 

I g{e)dpio^{2nr f 9{e)f{d)dd. 

Let Q <k <n and let H be an [n — A;)-dimensional subspace of R". Fix an 
orthonormal basis ei, . . . , in the orthogonal subspace H^. For a star body 
7^ in R", define the (n — k)- dimensional parallel section function^ Ak,h as a 
function on R*^ such that for ti G R^ 

A-k,h{'^) = Vol„_fc(X r\{H + uiBi H h UkSk}) 

= / x{\\x\\K)dx, 

J {xgR" : {x,ei)=ui,...,{x,ek)=Uk} 

where x is the indicator function of the interval [0, 1]. If is infinitely smooth, 
the function Ak,h is infinitely differentiable at the origin. We shall make use 
of the following fact: 

"'The parallel section function in this generality was first introduced in [f9], where the 
author gives a characterization of several classes of generalized intersection bodies in the 
language of functional analysis. The parallel section function and its further generalizations 
proved useful for the solution of virtually every generalization of the Buscmann-Petty prob- 
lem, e.g. [26, 37]. In particular, the parallel section function on was used in the Fourier 
analytic proof of the original Busemann-Petty problem, [12]. 
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Lemma 2. ([19], Theorem 2) Let K be an infinitely smooth origin symmetric 
star body in and < k < n. Then for every {n — k) -dimensional subspace 
H of W and for every m e N U {0}, m<{n-k) /2, 

where A denotes the Laplacian on W'. 

1.2.4. The Fourier and Radon Transforms of Re-invariant Functions. The fol- 
lowing simple observation is crucial for the application of Fourier methods to 
sections of convex bodies in the complex case. It translates the i^^-invariance 
of a body K into a certain invariance of the Fourier transform of its Minkowski 
functional raised to some power. We reproduce the proof of this observation 
here for its simple and insightful nature. 

Lemma 3. ([24], Lemma 3) Suppose that K is an infinitely smooth Re-inva- 
riant star body in M^". Then for every < p < 2n and ^ e S^"""^ the Fourier 
transform of the distribution is a constant function on S^"~^ HH^. 

Proof : As mentioned above the Fourier transform of H^^Hk^ is an infi- 
nitely smooth function on \ {0}. Since the function \\x\\k is -Re-invariant, 
by the connection between the Fourier transform of distributions and linear 
transformations, the Fourier transform of is also i?^i-invariant. Recall, 

from the introduction, that the two-dimensional space is spanned by two 
vectors ^ and Every vector in S^"~^r\H^ is the image of ^ under one of the 
coordinate- wise rotations Re, so the Fourier transform of is a constant 

function on S^*^"^ fl H^. 

□ 

Denote by C6((S^"~^) the space of i?e-invariant continuous functions on the unit 
sphere S^"^~^, i.e. continuous real- valued functions / satisfying /(^) = f{Re^) 
for any ^ e g2n-i ^^^^^ q ^ |-q^ 277] . The complex spherical Radon transform, 
introduced in [25], is an operator 7^e : (^^(S^"-^) ^(S^"-^) defined by 

7^J(0 = / f{x)dx. 

To derive a formula for the volume of the (2n — 2) -dimensional section of an 

i?0-invariant star body in R^", contained in the unit ball, by a hyperplane 
with respect to the volume clement in H^, wc use the following recently 
established connection between the complex spherical Radon transform and 
the Fourier transform. 

Lemma 4. ([25], Lemma 4) Let f G Ce(S^"~^) be an even function. Extend 
f to a homogeneous function of degree —2n -\- 2, f ■ 7-"^"+^, then the Fourier 
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transform of this extension is a homogeneous function of degree —2 on M^", 
whose restriction to the unit sphere is continuous. Moreover, for every ^ e 

g2n-l 

Lemma 5. Let K be a continuous Rg-invariant star body in M^" with n > 2 
contained in the unit ball. For ^ e S^"~^, we have 

Proof : We compute: 

dUn-i 



JK 



KnH^ 



jn— 1 



/" 

-'^nHf Jo 



\\k 



/s2"-inJ/j Jo (1 - r^)"" 
Using that \x\ — 1, we rewrite the above integral as: 



drdx 



HVol2„-2(i^ n H^) = 8"-i / / ^ / drdx . 

is2"-inH^ Jo U ~ J 

The function under the first integral sign is a homogeneous function of degree 
—2n + 2 and thus by the above lemma we obtain: 



□ 



|U||-2 

2. Connection with the distribution — " "'^ 



We now turn to our problem. First we construct a counterexample to the 
Busemann- Petty problem in HJ^ for n > 4. We use the same idea as in [33], 
namely that any Riemannian space is locally close to being Euclidean. 

Theorem 2. There exist Rg-invariant convex bodies K and L in M^** with 
n > 4 contained in the unit ball so that 

HVol2„-2(i^ n H^) < HVol2„-2(L n H^) 

for every ^ e S^"'"^, but 

HVol2„(X) > HVol2n(L) . 
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Proof : Let K and L be i?e-invariant convex bodies in M^" with n > 4 that 
provide a counterexample to the complex Busemann- Petty problem, see [24], 
i.e. 

Vol2„-2(i^ n H^) < Vohn-2{L n H^) (2) 
for every ^ G S^"~^, but 

Vol2„(i^) > Vol2n(L) . (3) 

We can and will assume that bodies K and L are infinitely smooth. Note that a 
dilation by a positive factor is an automorphism of invariant convex bodies 
in M^". Since inequality (3) is strict, we can dilate the body L by a positive 
factor greater than one, to make inequality (2) strict as well. Furthermore, 
there is an e > so that 

(1 + e)Vol2n_2(i^ n H^) < Yohn-2{L H H^) 

for every ^ G S^"~^, but 

VohniK) > (l + e)Vol2„(L). 

Here we are using the fact that ^ h- )■ Ak,h^{^) is continuous. It follows by an 
argument similar to one given in [22], Lemma 2.4. And any dilation of bodies 
K and L by a factor a > will also provide a counterexample. 

Choose a so small that both bodies aK and aL he in the ball of radius s 
that satisfies the inequality 

We use the same letters for the dilated bodies. Then for the volumes of bodies 
K and L we obtain: 



HVol2n(L) = 8 



dX2n 



(1 - 

< S'^fl + e) /" dX2n 



= 8"(l + e)Vol2n(L) 

< 8"Vol2n(i^) 



Jk 



dX2n 

K 

dX2n 



< 8"" 

- (1 - 

= HV0l2n(i^) , 
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and similarly, for the volumes of sections we have 

dX2n-2 



HVol2„-2(i^ni/^) = 8"-^ / 

JKnH^ I J- 

/ 

JKi 



2\n 



<8'^-i(l + e) / dx2 



2n-2 

'KnH^ 

, r,n-l f dX2n-2 

- Jl^h, (1 - 
^RYohn-2{LnH^). 

□ 

The connection between the Busemann-Petty problem in and the distri- 

I .11-2 

bution — y' y^^ 2 is the following: 

Theorem 3. The answer to the Busemann-Petty problem in is affirmative 
if and only if for every Rg-invariant convex body in M^" contained in the unit 

||^||-2 

ball the distribution — " |^ -2 is positive definite. 

This theorem will follow from the next two lemmas. 

Lemma 6. Let K be an Rg-invariant convex body in R^" contained in the unit 
ball such that — ^l^^^j^ .2 is a positive definite distribution on M^". And let L be 

an Rg-invariant star body in M^" contained in the unit ball so that 

HV0l2„-2(i^ n H^) < HV0l2n-2(iv H H^) 

for every ^ e S^""^. Then 

HVol2n(i^) < HVol2„(L) . 

2 

Proof : Observe that function is an increasing function on the interval 
(0, 1). We use this observation to estimate the following expression: 

^2 rb ^2n-3 ^2 rb ^2n-l n _ ^.2^ 

' dr = / r- — — — -ar 



1 - a2 y„ (1 - r2)" I -a? J a (1 - ^^)"+^ 

r-fe ^2n-l q2 (1 _ ^2) 



a (1 - r2)"+i 1 - a2 r 



2 



dr 



j.2n-l 
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where a, b are in (0, 1). Observe that the above inequahty it true in case a < b 
as well as in case b < a. Integrating both sides in the above inequality over 
the unit sphere S^"~^ with a — \\x\\]^^ and b — H^^H^''^ we obtain: 

r HNII-i ^2n-3 r ^2n-l 

/ 11 11 9 / 7" 7;^drdx < / 7" , ^ drdx . 

is^n-i 1 - ||x||^Vw-i (l-r2)n -7s2„-i4||-i (l-r2)-+i 

Next we show that the left hand side in the above expression is positive. This 
amounts to showing that 



I 



\T\l - iMk f ||'rl|-2 /-II^IIl^ „2n-3 



drdx < / — — ^ / — :r^drdx . 



Indeed, let d/io be the measure corresponding to the Fourier transform of the 
positive definite distribution — " |^ a , then using Corollary 1 we obtain: 

/■ IItII"^ /-II^IIk^ r2"-3 
^ ^ 7s2n-i 1 - Ikllx (l-r2)- 

^ 2n-3 



^2n-l 




;i-r2) 

= — ^ / mohn-2{KnH^)dtlo{^) 

27r /" 

<— ^ / HVol2„_2(Lni/5)rf/xo(C) 



-2n+2 



g2n-l 



(27r)2" / - 



\^\k 



Wi,^ „2n-3 



This implies that the right hand side in equation (4) is positive as well, which, 
in turn, shows 



I / T-, ^^T-rrdrdx < / 7- -—-drdx . 

That is, 

HVol2n(X) < HVol2„(L) . ^ 
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Next result is the inversion of the previous lemma. Its proof is based on a 
standard perturbation argument, see [18], Theorem 2, [36], Theorem 2, or [33], 
Proposition 3.9. 

Lemma 7. Suppose there is an infinitely smooth Re-invariant convex body 

K contained in the unit hall of R^" yjHii strictly positive curvature^ so that 

II ii~^ 

— "^ j^ i is not a positive definite distribution on R^". Then one can perturb 

the body K to construct another Re-invariant convex body L contained in S^"'"^ 
so that for every ^ e S^""^ 



but 



HVol2„(L) > W^ohniK) . 



Proof : It follows from our assumptions that the Fourier transform of 

— " 7^ i is negative on some open subset VL of the sphere S . iig-invariance 
i-( 

of the body K implies the i?g-invariance of the set Vt. Choose a smooth non- 
negative i?g-invariant function / on S^""^ with support of / contained in Vt and 

extend / to an i?e-invariant homogeneous function / of degree —2 

on M^" xhe Fourier transform of this extension is an i?^i-invariant infinitely 
smooth function on R^" \^ |o}, homogeneous of degree — 2n + 2, i.e. 

with g G C°^(S2"^^). Since / is i?e-invariant, so is g. Define an origin sym- 
metric body L contained in S^"~^ by 

' io (l- r^Y ' ' Jo (l- r^Y ''\\x\) ' 

for some e > 0. By a similar argument as in [36], Proposition 2, for small 
enough e, the body L is convex. Prom the i^^-invariance of K and g follows 
the it^e-invariance of L. Thus L is an i^^-invariant convex body contained in 



By strictly positive curvature we mean that the boundary of K does not contain any 

straight hnc segments. More precisely, the normal curvature of K at any point p on the 
boundary of K is strictly positive in any direction v, where v G TKp is any element in the 
tangent space to K at p, see [32] . 
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g2n-i Using Lemma 5, we compute: 



jn— 1 



RYohn-2{LnH^) = ^\\x 



-2n+2 



27r 
"2^ 



\x\ 



-2n+2 



r 



2n-3 



(1 -r2)" 



dr] (0 



(1 -H)" 



- 6(27r)^V 



< 



in— I 



27r 



-2n+2 



2n-3 



(1 - r^y 



:dr] (0 



HVol2n-2(i^ni/^). 



As in the proof of the previous lemma, to complete the proof it is enough 
to show the following: 



(27r) 



2n 



\X\ 



-2 fMl^' ^2n-3 
K 



1-2 



s2n-i 1 — ||a;||^ Jo 
( 

(27r)2" 



\x\ 



(1 -r2) 

,-2 ^ 

ft: 



-drdx 



q27i— i -I 



-2n+2 



I 



\x\ 



r 



2n-3 



(1 -r2)" 



dr dx 



-I 



\x\\l 



\ 



\X\ 



K 



\x\ 



(0 \A 



(0 \x\ 



-2n+2 



-2n+2 



1^1 

i^iIl r 



2n-3 



(1 - r^)'' 



:dr\ iOdC 



E(27r)2'^ / 



\x\ 



2n-3 



(1 - r2)" 



\X\ 



K 



x\\kJ 



dr] iOd^ 



iOfiOd^ 



( V 

' ll™ll-2 > 



> 



(0 \x\ 



is2"-i 1 _ ( _N_^ 
\Mk) 



2n-3 



(1 -r2)^ 



2 /■IIsIIk ^2n-3 
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□ 



3. Solution of the Problem 

In view of Theorem 2 we only have to find out whether the distribution 
— y i is positive definite for the complex dimension two and three, the 

case of the complex dimension one is trivial.^ The parallel section function, 
defined earlier, will help us to carry out this task. The idea is to express the 
distribution of interest in terms of the parallel section function. 

In dimension 2n, Lemma 2 with H = H^, ^ G S^""^, in which case k = 2, 
reads as follows: Let K be an infinitely smooth iig-invariant star body in M?^, 
then for m e N U {0}, m< n - 1 



Since the above integral is taken over the region S n if^, by Lemma 3, it 
follows that 



— 1 V 

' «^ ' 27r(2n-2m-2)^" "-^ ' ^ ' 



In complex dimension two there is only one choice for m, namely m = 0, and 
equation (5) becomes 

1 

47r' 

In complex dimension three, evaluating equation (5) for m = 1, we obtain 

Lemma 8. For any Re-invariant star body K in contained in the unit hall 
the distribution — is positive definite. 

Proof : Assume first that K is an infinitely smooth ii^j-invariant star body 
in W^. Define another body M by 

II 11-2 W-^Wk 
Wxy — 



\M 



Since K is an i?g-invariant star body, so is M. Note also that the denominator 
in the defining expression for M is never zero, since the body K is contained 



^Indeed, -Rg-invariant convex bodies in C = are closed discs around the origin. 
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in the open unit ball. This implies that the body M is also infinitely smooth. 
Hence 

AM,HM = ^mMni). 

and consequently is positive definite. Thus for any infinitely smooth 

_Re-invariant star body K the distribution — ' 2 is positive definite. And 

therefore, by the approximation results in Section 1.2.2, this is true for any 
i?6(-invariant star body. 

□ 



Lemma 9. There is an infinitely smooth Re -invariant convex body K in R 
contained in the unit ball for which the distribution — " |^ ^ is not positive 

definite. 



Proof : For an element ^ of R^, ^ = (61,62,61262,61,62), denote by 
6 = (61, 62) and by I = (61, 62, 61, 62), then ^ = (6 6)- We will work with 
the following map, written in polar coordinates as 



^'^)^ \/^^^'^ ■ (6) 




Note that this map, restricted to the two-dimensional plane xy, takes the line 
x = i to the hyperbola {a? — — = 1, provided that — 1 > 0, and it 
takes the ellipse + (1 + = 1 to the line y — Denote the equation 

of the elliptic arc above the x-axis by e, i.e. e{x) — and the equation 

of the hyperbolic arc to the right of the y-axis by h, i.e. h{y) — \J if^. Now 
define a convex body K in by 

K = {ie^' : |6<-and |6| < e(|6)} • 

a 

We restrict the values of h to be strictly greater then one, this ensures that the 
body K is contained in the unit ball. As before, define a star body M by 

\x\'~'^ ' 



\^\K 



\M\k) 



^Indeed, writing the equation of the hne a; = i in polar coordinates, we obtain y_ 



cos"^ e-1 ' ^'^ ^^"^ image of the line is = ^2 cpsS g_i ■ Similarly to find the image of the 
ellipse, we write its equation in polar coordinates: = j^i^r^^^- Then = g and 

hence the image of the ellipse is r = • 
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The body M is an image of the body K under the map (6) and hence it can 
be described as 

M = {e e : lel < /ideal) with 1^31 < \} . 

The bodies K and M wc constructed arc not infinitely smooth. However, 
since we can approximate the body K in the radial metric by a sequence of 
infinitely smooth convex bodies, there is an infinitely smooth convex body K' 
that differs from K only in an arbitrary small neighborhood of the boundary 
of K. This modification of K will make the body M infinitely smooth as well. 
We use the same letters K and M for these modified bodies. By construction 
both bodies arc i^g-invariant. 

Let X = (XjXs) G M with x^ ^ (0,0). Choose { G in the direction 
of X3. Fix an orthonormal basis {61,62} for E.^. For G with \u\ > ^ 
Am,h^{u) — 0, and otherwise 

Am,h^ {u) = Vol4(M n {H^ + uiei + 'U262}) 

x{\\x\\M)dx 



{xeR^" : {x,ei)=ui,{x,e2)=U2} 

f rK\u\) 
/ / r^drde 
Js3 Jo 

,C3,MI«I)' 



where |S"^^| stands for the surface area of the unit sphere S"""^ in M", i.t 
27r5/r(f ). Setting a = 2, wc get 

Am,h.{u) = — I ) and consequently AAm,hAu) = (1 + 2|m|^) 



2 V 3 / 
Since M is infinitely smooth we have 



1-2 



This shows that ( — "'^''^ 2 1 (C) — (Ik II a/) '^(C) is negative for some direction 

□ 

Now we are ready to prove the main result of this paper: 
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Theorem 4. The answer to the Bus emann- Petty problem m the complex hy- 
perbolic n-space, Hg, is affirmative for n <2 and negative for n > 3. 

Proof : By Lemma 8, the distribution — " /5 -s^ is positive definite for any 

i?0-invariant convex body K in M^, as any such body is a star body. The 
affirmative answer for n = 2 now follows from Theorem 3. 

For n > 4 the negative answer was provided in Theorem 2. For n = 3, 
by Lemma 9 there is an infinitely smooth it^^-invariant convex body K in 

||3,||-2 

contained in the unit ball for which the distribution — ' 7^ is not positive 

definite. Observe that we can assume that the body K has strictly positive 
curvature by applying a standard perturbation trick, namely by setting 

for some e > small. The negative answer now follows from Lemma 7. 

□ 
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